A Multi-Agent System for Distributed Information Retrieval on the World Wide Web

In this paper a novel approach concerned with the general framework of Information Management, is presented. We use a Multi-Agent System to cope with the problem of Distributed Information Retrieval. The Distributed Information Retrieval task deals with the collection of information from multiple and usually heterogeneous information sources that exist in a distributed environment, which in our case is the World Wide Web. 1 Overview of the system's architecture The advent of large wide-area networks, Internet is the most characteristic example, has caused a vast increase both in the information availability and in the number of the information sources. This evolution offers great promise for obtaining and sharing diverse information conveniently. However, the multitude, diversity and the dynamic nature of on-line information sources make accessing any specific piece of information an extremely difficult task. One way to address these issues is to use information agents. These Distributed Information Retrieval agents should be able to: ƒ accept a request from a human or agent client, ƒ translate this request into a language understood by the information sources, ƒ identify the information sources that contain information relevant to the request, ƒ pose the request to these sources, ƒ collect the corresponding results, ƒ process the returned results and ƒ present the results to the client. We have followed this approach in developing our information retrieval system for the WWW. The overall agent architecture is as follows (see Figure 1). The inter-agent communication is based on standard Knowledge Query Manipulation Language (KQML) performatives [Patil 94]. Our system supports a collection of information sites. The notion of an information site is used to describe a logical entity that contains a set of information sources. It is a logical clustering of actual-physical WWW sites. In each information site, we find the extractor agent and the information source agent. The extractor periodically scans through all the information sources, represented as URLs. These can be URLs of the top-level web pages of various research groups, for example. The extractor traverses through all the local documents (e.g. documents belonging to that research group) that are accessible via a chain of links from the top-level page. It classifies each such page as 'interesting' or not and extracts from each 'interesting' web page the key features and represents these features in a relational/attribute-based form. For example, it will describe an identified research paper in terms of attributes like authors, title, topics, keywords, document location (URL), abstract of document location (URL) and referenced authors.